Interview Playbook [Systems Design]

Clarifying questions to keep in your pocket

Common traps

• Boxes before requirements. A perfect design for the wrong problem scores zero.
• Name-dropping without why. "I'd use Kafka" is nothing; "a queue decouples the spike from the DB, Kafka because we need replay" is the bar.
• Uniform shallowness. Covering ten topics at 1-inch depth reads L4. Two dives at real depth reads L5.
• Premature scale. Don't shard a database that fits on one machine. Start simple, scale when the numbers demand it — and say that's what you're doing.
• Ignoring hints. Interviewer nudges are the rubric leaking. Take them.
• Silence while drawing. Narrate every box as you add it.

What they're scoring at L5

• Ownership: you drove start to finish; the interviewer only steered.
• Requirements → design traceability: every choice ties back to a stated constraint.
• Trade-off fluency: two options and a justified pick, every time.
• Quantified reasoning: rough numbers that lead to decisions.
• Depth on demand: at least two areas where you go genuinely deep.
• Pragmatism: simplest thing that meets the requirements, with a scaling path.

Full-stack edge — use it

• API contract details (pagination, idempotency, errors) — most backend candidates skip these; you shouldn't.
• Client-side story: optimistic updates, browser caching, stale-while-revalidate.
• Real-time delivery trade-offs (WebSocket vs SSE vs polling) end to end, client included.
• When infra depth runs out, say so honestly and reason from first principles — that beats bluffing.

Technology overview — Sharding

• What: split one dataset across nodes by a key (hash or range on user_id).
• Use when: writes or data size exceed one primary — not before. Say that out loud.
• Trade-offs: cross-shard queries become fan-outs; joins and transactions get hard; hot keys (celebrities) skew load; resharding is painful — pick the key carefully, and colocate rows that commit together so writes stay single-shard (no 2PC).

SQL vs NoSQL

• SQL (Postgres, MySQL): joins, ACID transactions, strong consistency. The default — most systems fit one instance + read replicas.
• NoSQL (DynamoDB, Cassandra): key-based access at massive write scale, built-in horizontal scaling. Cost: no joins, limited transactions, query patterns fixed up front.
• Redis: in-memory KV — caching, sessions, counters, rate limits. Sub-ms reads, but RAM-priced and lossy on crash.

Durable message queues

• What: persist work between producer and consumer, decoupling spikes from the DB.
• Kafka: replayable append-only log, huge throughput — event streams, fan-out, analytics pipelines.
• SQS / RabbitMQ: simpler task queues with per-message ack — background jobs, emails, thumbnails.
• Trade-offs: at-least-once delivery → consumers must be idempotent; ordering only per partition; adds latency and ops surface.

Object storage (S3)

• What: cheap, ~11-nines-durable blob store for images, video, logs, backups.
• Use when: anything large or unstructured — store the blob in S3, its URL in the DB, serve via CDN with presigned URLs for uploads.
• Trade-offs: higher latency than a DB, no queries or partial updates — wrong for small hot records.

CDC + outbox — commit to DB, then emit to Kafka

• Problem (dual write): you need to update the DB and publish an event (Kafka). Doing both directly isn't atomic — DB commits but the publish fails (or vice versa) and the two diverge.
• Outbox: in the same transaction as the business write, insert the event into an outbox table. One atomic commit — data and intent-to-publish succeed or fail together.
• CDC relay: a separate process (Debezium tailing the WAL, or a poller) reads committed outbox rows and publishes to Kafka. DB is the single source of truth; the event is guaranteed to ship.
• Cost: at-least-once delivery → consumers must be idempotent; small publish lag.
• Line: "To dodge the dual-write problem I write an outbox row in the same txn and a CDC relay publishes to Kafka — atomic commit, at-least-once, consumers dedupe."

Load balancer + API gateway — the edge

• Load balancer: spreads traffic across stateless app servers with health checks (drops dead nodes). L4 (TCP, fast) vs L7 (HTTP, can route by path/header). Stateless tier is what lets it scale horizontally.
• API gateway: the single client entry point and policy layer — auth, rate limiting, routing to services, TLS termination, request shaping. In microservices it fronts many services so clients hit one endpoint.
• Where they sit: Client → CDN → LB → API gateway → app/services. LB = spread load, gateway = policy + routing; often merged at the edge.
• Line: "LB load-balances a stateless app tier with health checks; the gateway handles auth, rate limiting, and routing so services stay focused on business logic."

Connection pooling — short

• What: reuse open TCP connections instead of one per request — amortizes TCP + TLS handshake RTTs.
• Use when: repeated calls to the same long-lived backend (app→DB, app→Redis). Size to the backend's connection cap (Postgres limits are low → PgBouncer).
• Note: avoids the handshake, not the round trip — say that. Skip for HTTP/2-3, serverless, or many distinct hosts.
• Line: "Pool connections to Redis/DB to amortize handshake cost, sized to the connection limit, plus pipelining to batch round trips."

ACID + shard to colocate transactions

• Postgres gives you ACID: atomic multi-row commits with real isolation. That's why booking/payment logic stays simple — flip N tickets and write the booking in one transaction, all-or-nothing.
• Shard so a transaction stays on one shard. Pick the key so rows that commit together colocate — e.g. shard by eventId so every ticket for an event lives on the same shard.
• Why it matters: a cross-shard transaction needs distributed commit (2PC) — slow, complex, and a new failure mode. Single-shard transactions keep ACID cheap.
• Line: "I'll shard by the entity we transact against so writes stay single-shard and I never need 2PC."

Idempotent writes — safe retries

• Client generates a UUID per logical write, sent as Idempotency-Key. The client owns it because it needs to be stable across retries.
• Server stores key → response on first execution; a retry with the same key returns the stored response instead of re-running the write.
• Why: networks retry on timeout — without this you double-charge or double-book. Essential on any non-idempotent POST; forward the key to the payment provider so it dedupes the charge too.
• Line: "Every mutating call carries a client UUID; a retry returns the same response, so a timeout never double-applies."

Cache strategy + stampede protection

• Cache-aside (default): app reads cache, on miss reads DB and populates. On write, update the DB then invalidate the key. Simple, resilient to cache loss.
• Write-through / write-back: writes go via the cache — stronger read consistency (through) or faster writes at durability risk (back). Only reach for these with a reason.
• Stampede (thundering herd): a hot key expires and thousands of misses hit the DB at once. Fix with jittered TTLs, a single-flight lock (one request recomputes, others wait), and refresh-ahead before expiry.
• Line: "Cache-aside with jittered TTLs and a single-flight lock, so one hot key expiring can't stampede the DB."

Consistent hashing

• What: place keys and nodes on a ring; a key maps to the next node clockwise. Adding/removing a node moves only ~1/N of keys, not the whole keyspace.
• Use when: sharding caches or DBs across a node set that changes — autoscaling, failures, adding capacity. Virtual nodes smooth out uneven load.
• Why it matters: naive hash % N remaps almost everything when N changes, cold-flushing every cache at once.
• Line: "Consistent hashing so adding a shard remaps ~1/N of keys instead of reshuffling the whole ring."

Replica lag + read-after-write

• Setup: leader takes writes, replicas stream asynchronously and lag by ms to seconds. Great for scaling reads, but the copy is a moment in the past.
• Read-after-write trap: user writes, immediately reads a lagging replica, and doesn't see their own change. Fix by routing that user's reads to the leader briefly, or a version/timestamp token, or sticky routing.
• Correctness rule: never authorize a write off a replica (e.g. "is this seat free?"). That read must hit the leader with a row lock — replicas are for display, not for gating writes.
• Line: "Replicas lag, so I route read-after-write to the leader and never make a write decision from a replica."

Backpressure + dead-letter queue

• Backpressure: a queue decouples spikes, but consumers fall behind. Bound the queue and shed/throttle producers so you degrade instead of OOM-ing. Queue depth is a key metric + alert.
• Retries + DLQ: transient failures → retry with exponential backoff; after N tries a poison message goes to a dead-letter queue for inspection instead of blocking the pipeline.
• Idempotent consumers: delivery is at-least-once, so processing the same message twice must be safe.
• Line: "Bounded queue with backoff retries and a DLQ for poison messages; consumers idempotent because delivery is at-least-once."

Sagas + compensating actions — when one transaction won't fit

• What: a workflow that spans services or an external system (booking + Stripe, order + inventory + shipping) can't sit in one ACID transaction. Break it into a sequence of local transactions, each with a compensating action that semantically undoes it if a later step fails. This is the alternative to 2PC.
• Compensation is a semantic undo, not a rollback: refund reverses a charge, release reverses a reservation, cancel reverses a shipment. It must be idempotent, since it may be retried.
• Ticketmaster example: reserve seats (compensate: release) → charge Stripe (compensate: refund) → confirm booking. Charge fails → release the seats; a crash after charge but before confirm → retry confirm, or refund if it can't complete.
• Choreography vs orchestration: choreography = each step emits an event (via the outbox) and the next reacts — decoupled, simple; orchestration = a central coordinator drives the steps — clearer for complex flows, one place to reason about state.
• Trade-off (say it): a saga has no isolation — intermediate states are visible (a seat is reserved-but-not-paid). You design for that window explicitly, which is exactly what the reservation TTL / reserved_until is for. Consistency is eventual.
• Line: "The booking-plus-payment flow is a saga, not a distributed transaction: reserve, charge, confirm, each with a compensating action — release and refund — and idempotent so retries are safe."